The same estimator scores 1.587e-7 on AIcrowd's 50 leaderboard MLPs and
2.235e-7 on 512 MLPs of the public full split. The typical MLP is
equally hard in both — the medians differ by 17%. What the public 50 don't have is the
upper tail: 11.3% of full MLPs sit beyond the leaderboard set's entire
range, and they carry 35.7% of its total error.
Grader: sub 325219, GRADED, 50/50Local: HF v1-phase1, flopscope 0.10.0Same code: F/B 0.2669 vs 0.2665Measured: 2026-08-10
Adjusted score, AIcrowd grader
1.587e−7
50 leaderboard MLPs · submission 325219
Adjusted score, full split
2.235e−7
512 MLPs · analytic FLOPs only
Median per-MLP MSE
17%
apart: 4.93e−7 vs 5.77e−7 — the typical MLP is the same
Full-split MLPs beyond the public 50's entire range
11.3%
58 of 512, carrying 35.7% of total MSE
Figure 1 · per-MLP final-layer MSE
Three samples of the same estimator, one shared log axis
Each panel is one MLP sample, binned at ⅛-decade resolution and drawn as a share of that
sample's own MLPs so the three are comparable despite differing n. The
rug beneath each panel is one tick per MLP. The shaded band marks everything beyond the
largest MSE the leaderboard set contains (1.534e-6) — empty in the top panel by
construction, populated in both local samples.
Public 504.22e−6 – 5.62e−6 0 MLPs · 0.0% of 50
Bin width ⅛ decade. Vertical rules mark each sample's median
(solid) and mean (dotted) — the distance
between them is the skew; exact values are in the table below. Hover any bin for its range and count.
Figure 2 · could the public 50 be an ordinary draw?
Resampling 50 MLPs from the full split, 20,000 times
If the leaderboard's 50 MLPs were a plain random sample of the same population, their mean MSE
would land inside this distribution. It doesn't: 0.19% of draws come in at or below
the observed 5.757e-7. That is the same conclusion Line P reached by a different route, and it holds
for the current estimator.
1.131e−6 – 1.150e−6119 of 20,000 draws
20,000 bootstrap resamples (with replacement) of 50 MLPs drawn from the
512-MLP full-split run. Seed 0.
Table · every number in the figures
Distribution statistics
Sample
n
Adjusted score
Mean MSE
Median
p90
p99
Max
Max ÷ median
Skew
Top 10% share
Beyond 1.534e-6
Public 50
50
1.587e−7
5.757e−7
4.925e−7
1.006e−6
1.524e−6
1.534e−6
3.1×
1.08
23.1%
0 (0.0%)
Full split
512
2.235e−7
8.459e−7
5.767e−7
1.653e−6
4.120e−6
9.606e−6
16.7×
4.30
33.2%
58 (11.3%)
Mini split
100
2.338e−7
8.944e−7
5.409e−7
1.399e−6
4.044e−6
1.533e−5
28.3×
7.74
40.0%
9 (9.0%)
Adjusted score is mean(mse × max(0.1, C/B)). The grader row
uses its own C including residual wall time; the local rows use analytic
FLOPs only (F/B), since residual time on a shared 192-core node is a
property of the node, not the estimator.
Reading it
What follows, and what doesn't
The estimator is not regressing. Median per-MLP MSE is 4.93e-7 on the
leaderboard set against 5.77e-7 on full — a 17% spread that is ordinary sampling
noise between two 50–512-MLP draws. Typical MLPs behave the same in both places.
The gap is carried by ~11% of MLPs. Delete the 58 full MLPs that
lie beyond the leaderboard set's maximum and the mean falls from 8.459e-7 to
6.131e-7, within 6% of the public 50's 5.757e-7.
This is why N=32768 is still deployed. The full split says MSE falls as
N−1.00 with no bias floor, so more points keep paying; the public 50 fit a bias floor
of 1.31e-7 and rejected N=49152 by 9%. Tail-heavy MLPs are variance-dominated — the two sets
disagree precisely because one has the tail and one doesn't.
The prize re-evaluation is drawn from the full distribution, which is the one
with the tail. That makes ~2.2e-7 the number to optimize against and ~1.59e-7 the number the
leaderboard happens to show.
Caveats worth holding: the two sources use different ground truth (the grader's own
Monte Carlo versus the HF-baked final_means) on disjoint MLP seeds, so
they are not paired; and 512 of the 1000 full-split MLPs were run, not all 1000. Neither affects
the shape comparison, which is what the figures are about.